Stochastic Difference of Convex Algorithm and its Application to Training Deep Boltzmann Machines
نویسندگان
چکیده
Difference of convex functions (DC) programming is an important approach to nonconvex optimization problems because these structures can be encountered in several fields. Effective optimization methods, called DC algorithms, have been developed in deterministic optimization literature. In machine learning, a lot of important learning problems such as the Boltzmann machines (BMs) can be formulated as DC programming. However, there is no DC-like algorithm guaranteed by convergence rate analysis for stochastic problems that are more suitable settings for machine learning tasks. In this paper, we propose a stochastic variant of DC algorithm and give computational complexities to converge to a stationary point under several situations. Moreover, we show our method includes expectation-maximization (EM) and Monte Carlo EM (MCEM) algorithm as special cases on training BMs. In other words, we extend EM/MCEM algorithm to more effective methods from DC viewpoint with theoretical convergence guarantees. Experimental results indicate that our method performs well for training binary restricted Boltzmann machines and deep Boltzmann machines without pre-training.
منابع مشابه
A Hybrid Algorithm based on Deep Learning and Restricted Boltzmann Machine for Car Semantic Segmentation from Unmanned Aerial Vehicles (UAVs)-based Thermal Infrared Images
Nowadays, ground vehicle monitoring (GVM) is one of the areas of application in the intelligent traffic control system using image processing methods. In this context, the use of unmanned aerial vehicles based on thermal infrared (UAV-TIR) images is one of the optimal options for GVM due to the suitable spatial resolution, cost-effective and low volume of images. The methods that have been prop...
متن کاملInductive Principles for Learning Restricted Boltzmann Machines (DRAFT: August 25, 2010)
We explore the training and usage of the Restricted Boltzmann Machine for unsupervised feature extraction. We investigate the many different aspects involved in their training, and by applying the concept of iterate averaging we show that it is possible to greatly improve on state of the art algorithms. We also derive estimators based on the principles of pseudo-likelihood, ratio matching, and ...
متن کاملTraining restricted Boltzmann machines: An introduction
Restricted Boltzmann machines (RBMs) are probabilistic graphical models that can be interpreted as stochastic neural networks. They have attracted much attention as building blocks for the multi-layer learning systems called deep belief networks, and variants and extensions of RBMs have found application in a wide range of pattern recognition tasks. This tutorial introduces RBMs from the viewpo...
متن کاملSequential Labeling with online Deep Learning
In this paper, we leverage both deep learning and conditional random fields (CRFs) for sequential labeling. More specifically, we propose a mixture objective function to predict labels either independent or correlated in the sequential patterns. We learn model parameters in a simple but effective way. In particular, we pretrain the deep structure with greedy layer-wise restricted Boltzmann mach...
متن کاملAn augmented efficient backpropagation training strategy for deep autoassociative neural networks
We introduce Augmented Efficient BackProp, a strategy for applying the backpropagation algorithm to deep autoencoders, i.e. autoassociators with many hidden layers, without relying on a weight initialization using restricted Boltzmann machines (RBMs). This training method, benchmarked on three different types of application datasets, is an extension of Efficient BackProp, first proposed by LeCu...
متن کاملذخیره در منابع من
با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید
عنوان ژورنال:
دوره شماره
صفحات -
تاریخ انتشار 2017